1. Field of the Invention
The present invention relates to a method for providing digital images with a portable complex device equipped with a digital photography function (e.g. a digital camera or a mobile communication terminal having a camera module), and more particularly to a method for capturing a panorama mosaic photograph.
2. Description of the Related Art
As generally known in the art, digital photography devices obtain images focused at the focus length of the lens, The view angle of the obtained images ranges between 30-50° (in the case of a conventional camera), which is less than that of human's eye view, which ranges between 150-200°. A panorama mosaic photograph is obtained by the digital camera by photographing a number of scenes and reconstructing them into a large image by connecting the senses.
In order to construct a panorama mosaic photograph, a sequence of the captures images must overlap each other. By using the overlapping regions, the images are projected/transformed onto the same planar or curved surface and connected to each other. This process is followed by what is known as a stitching process for determining the boundary between images and a blending process for rendering the luminance and feel of color of the boundary region to feel natural. As such, implementation of a panorama image requires computer vision and image processing techniques including a geometric camera projection model, feature point and correlation extraction, projection transformation matrix estimation and image transformation, image boundary region estimation, and image blending.
Although projection transformation based on camera models and geometric projection is commonly required by all mosaic algorithms, there are a number of variations on the process of estimating projection transformation and on the stitching as well as blending techniques.
In order to estimate a projection transformation matrix between images, it is necessary to extract feature points from images, find the correlation between images, and estimate the transformation matrix. The feature points extracted in this regard may be edges or corners as in the case of conventional methods. It is also possible to directly derive the correlation based on a motion estimation technique, such as a block matching. Recently developed SIFT (Scale Invariant Feature Transform) is an excellent feature point extraction technique capable of deriving the correlation and transformation relationship between images in a more accurate and stable manner. However, such feature extraction and projection transformation estimation require a large amount of calculation and floating-point operation, and are unsuitable for real-time implementation in systems with limited operation capability, such as portable terminals.
Once a transformation matrix between images is obtained, respective images are re-projected onto the same mosaic plane or curved surface. Mosaic images are commonly projected onto a cylindrically curved surface. This is because, when a panorama image is to be created, a sequence of obtained images rotate with regard to the direction of the camera and constitute a cylindrical structure. A relationship for projecting a planar (two-dimensional) image onto a cylindrically curved surface in a three-dimensional space is defined by equation (1) below.
                                                        (                                                f                  ⁢                                                                          ⁢                                      X                                                                                            R                          2                                                -                                                  X                          2                                                                                                                    ,                                  f                  ⁢                                                                          ⁢                                      Y                                                                                            R                          2                                                -                                                  X                          2                                                                                                                                )                        ⁢                                                  ⁢                          (                              X                ,                Y                            )                                ∈                      Z            2                          ,                            (        1        )            
wherein f refers to the focal length, and R refers to the radius of the cylindrically curved surface. In most cases, the focal length and the radius have similar values. FIG. 1 shows a process for projecting a two-dimensional image onto a cylindrically curved surface.
When projected onto the same curved surface, images overlap each other. In this regard, how to process the overlapping portions (i.e. stitching technology) is crucial to panorama mosaics. There are two types of approaches: according to the first one, two overlapping images are blended properly, and, according to the second one, the boundary of two images is determined at the overlapping portion so as to differentiate between both images. More particularly, when two images are blended according to the first approach, the values of both images, which are superimposed on the same pixel, are averaged based on a weight so as to obtain a panorama image. In this case, blurring of images occur in the overlapping region. When the boundary of images is determined in the overlapping region according to the second approach, the optimum pixel path for naturally connecting the boundary of both images is searched for. After determining the boundary of images, a blending process for alleviating rapid change in luminance and color of both images is necessary. In general, a linear transparency is defined with reference to the boundary of images so as to blend two images. In order to correct the exposure of entire images, it is also possible to blend images based on different degrees of transparency for multiple frequency bands.
As mentioned above, the conventional panorama mosaic algorithms provides an accurate projection transformation matrix by extracting feature points and corresponding points between images. For more accurate image transformation, a process for optimizing the projection transformation matrix follows. However, the conventional algorithms have a drawbacks in that, although there is no problem in dealing with the above-mentioned processes on a software basis in conventional computer environments, real-time implementation of such a mosaic process with limited hardware, such as a portable terminal, is impractical when the current system level is considered. This is because complicated floating-point operations are necessary, together with repeated operations and memories for optimization. As such, although existing panorama mosaic algorithms exhibit excellent performance on a software basis, they are hardly applicable to popular digital cameras and portable terminals.
Another drawback is that, when it comes to the feature point and correlation extraction process commonly employed by existing algorithms, the projection transformation matrix cannot be obtained from the correlation unless suitable feature points are extracted between images. This means that the panorama process cannot be proceeded at all. As such, the existing algorithms have a common limitation of instability, i.e. their panorama results vary depending on the contents of images due for panorama.